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Abstract 


This  work  was  carried  out  for  The  US  Office  of  Naval  Research  under  NICOP  grant  N000 14-99- 1-1044.  This  final 
report  summarises  the  work  completed  in  the  first  phase  of  a  three  year  study  to  identify  the  factors  that  determine 
the  effectiveness  of  sonar  and  target  motion  analysis  operators  in  United  States  Navy  (USN)  and  UK  Royal  Navy 
(RN)  submarines.  The  aim  of  the  programme  is  to  highlight  best  practice  in  operator  performance  in  a  way  that  will 
contribute  to  training  effectiveness.  This  report  describes  the  progress  made  by  the  UK  in  preparing  for  this  study 
and  in  implementing  it.  A  parallel  study  of  USN  operators  is  being  conducted  by  personnel  from  the  Naval  Undersea 
Warfare  Center  (NUWC),  Newport.  A  comparative  Cognitive  Task  Analysis  will  be  a  principal  feature  of  the  study. 

As  planned,  the  first  phase  of  the  UK  work  programme  consisted  mainly  of  programme  set-up  and  of  developing 
data-collection  and  analysis  procedures  that  will  facilitate  effective  comparison  of  UK  and  US  findings.  It  also 
includes  familiarisation  with  the  technical-domain  and  initial  data-collection. 

An  extensive  programme  of  visits  and  data-collection  sessions  has  been  agreed.  Common  approaches  to  data 
collection  and  analysis  have  been  agreed  and  are  being  developed.  Data  are  being  exchanged.  US  researchers  have 
been  able  to  visit  RN  establishments  and  to  participate  in  some  of  the  interviews.  Analysis  of  technical  and  training 
materials  has  begun.  Factors  critical  to  success  are  being  elicited.  Detailed  data  collection  from  the  Royal  Naval 
Submarine  School  and  from  the  School  of  Maritime  Operations  has  begun.  Cognitive  task  analysis  has  begun  with 
both  experts  and  novices.  Some  early  results  are  described.  The  programme  of  future  work  is  also  described. 
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Executive  Summary 


El  Aims 

El.l  This  report  is  the  second  deliverable  due  under  NICOP  agreement  N000 1 4-99- 1  - 1 044  of  the  US  Naval 
International  Co-operative  Opportunities  in  Science  and  Technology  Programme  (NICOP).  The  customer 
for  this  work  is  the  US  Office  of  Naval  Research.  This  report  summarises  UK  progress  in  the  first  stage  of 
a  joint  US/UK  programme  to  compare  UK  and  US  sonar  and  Target  Motion  Analysis  operators.  It  is  believed 
that  there  may  be  systematic  performance  differences  between  the  operators  in  these  two  nations:  the 
programme  will  investigate  those  differences  and  suggest  ways  of  maximising  performance. 

E2  Programme  Design  and  Set  Up 

E2.1  An  important  part  of  this  stage  of  work  was  to  develop  a  co-ordinated  programme  design  that  would 
provide  the  research  resources  and  the  access  to  Royal  Naval  (RN)  facilities  that  are  required  to  meet  the 
objectives  of  ONR  and  the  NUWC.  Essential  administrative  issues  such  as  authorisation,  progress-review, 
reporting,  security  and  intellectual  property  rights  have  been  addressed.  The  agreed  working  procedures  are 
described  and  the  range  of  DERA,  MoD  and  RN  facilities  that  can  be  expected  for  the  execution  of  this 
programme  are  identified.  Arrangements  for  liaison  and  co-ordination  with  NUWC  and  ONR  are 
described.  The  important  process  of  developing  and  harmonising  the  experimental  methods  to  be  used  by 
NUWC  and  DERA  has  been  addressed.  These  are  described  and  will  be  continually  reviewed  and  updated. 

E3  Methods  for  Analysing  Operator  Performance 

E3.1  The  study  will  include  several  levels  of  analysis,  including  organisational  issues  (from  tactics  to  career 
patterns),  technical  issues  (e.g.  levels  of  automation  and  interface  design),  physiological  measures  (heart 
rate  and  eye-movement)  and  performance  measures  (process  and  outcome).  A  comparative  Cognitive  Task 
Analysis  (CTA)  will  be  a  principal  feature  of  the  study.  Applied  Cognitive  Task  Analysis  (ACTA.)  is  being 
used  as  a  common  tool  in  both  the  US  &  UK  studies.  Later  work  will  include  observations  of  experts 
operating  in  teams,  performance  measurement  and  physiological  measurement. 

E4  Analysing  Operator  Behaviour 

E4.1  Background  information  describing  the  training  and  development  of  operator  skills  and  experience  is  being 
collected  and  training  materials  are  being  examined  in  detail.  The  way  that  operator  functions  are  allocated 
within  the  command  team  is  being  explored:  operating  procedures  and  team  processes  are  being  analysed. 
Using  ACTA,  interviews  are  being  conducted  with  individual  trainees  and  experts  to  identify  the  cognitive 
skills  and  the  cues  and  strategies  that  are  used  in  sonar  and  TMA  tasks.  Later  work  will  focus  on  common 
tasks  and  scenarios  and  will  address  performance  measures  and  physiological  measures. 

E5  Factors  Critical  to  Performance 

E5.1  Work  has  begun  on  identify  ing  the  factors  that  are  critical  to  performance.  The  experience  of  the  operators 
and  the  team  processes  that  are  used  to  exploit  this  are  believed  to  be  important  factors.  Ways  of  analysing 
team  processes  and  of  measuring  performance  are  therefore  being  developed. 

E6  The  Next  Stages 

E6.1  A  common  situational  interview  will  be  developed  to  standardise  the  focus  of  UK  and  US  CTAs  and  to 

facilitate  comparisons.  The  allocation  of  functions  within  the  command  team  will  be  further  clarified. 
CTAs  will  continue  and  will  be  used  to  identity  each  individual’s  contribution  within  the  team.  Objective 
measures  of  performance  and  physiological  measures  will  be  derived.  Plans  for  the  public  dissemination  of 
the  initial  results  of  the  study  are  identified. 

ET  Conclusions  and  Recommendations 

E7. 1  Programme  set-up  has  been  completed.  Collection  of  background  data  has  begun  and  the  cognitive  task 
analysis  of  experts  and  novices  has  begun.  It  is  concluded  that  team  processes  may  be  a  significant 
performance  factor.  Methods  to  analyse  these  are  being  developed  and  it  is  recommended  that  full  use 
should  be  made  of  the  extended  UK  facilities  on  offer  in  the  next  phase  of  this  programme. 
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1  Introduction 

1.1  Milestone  Statement 

111  Thjs  work  was  carried  out  for  The  US  Office  of  Naval  Research  under  grant  N000 14-99- 1-1044  of  the 
Naval  international  Co-operative  Opportunities  in  Science  and  Technology  Programme  (NICOP).  This  is 
the  first  phase  of  a  three  phase,  three  year  study  to  identify  the  factors  that  determine  the  effectiveness  of 
sonar  and  target  motion  analysis  operators  in  United  States  Navy  (USN)  and  UK  Royal  Navy  (RN) 
submarines.  The  Terms  of  Reference  of  this  study  are  given  at  Appendix  A.  The  aim  of  the  programme  is 
to  highlight  best  practice  in  operator  performance  in  a  way  that  will  contribute  to  training  effectiveness.  The 
report  describes  the  progress  made  by  the  UK  in  preparing  and  conducting  a  cognitive  task  analysis  of  sonar 
operators. 

1.2  Background 

1.2.1  Opportunities  are  rare  for  conducting  a  detailed  cognitive  task  analysis  of  submarine  operator  performance. 
Most  endeavours  in  this  area  are  driven  by  the  design  of  new  systems  or  by  training  needs  analysis.  This 
mav  be  the  first  time  that  an  international  comparison  of  expert  performance  has  been  conducted  with  a 
view  to  capturing  best  practice  and  to  optimising  performance.  There  are  many  distinct  differences  between 
the  two  services  that  can  affect  the  performance  of  USN  and  RN  submarine  operators  (these  include  the 
submarines,  the  sensor  and  analysis  systems,  the  personnel  and  the  operating  procedures).  Experimental 
approaches  that  are  sensitive  to  the  similarities  and  difference  have  to  be  developed. 

1.3  Aims 

1  3  1  The  aim  of  the  programme  is  to  compare  UK  and  US  sonar  and  Target  Motion  .Analyst  (TMA)  operators.  It  is 
believed  that  there  may  be  systematic  performance  differences  between  the  operators  in  these  two  nations:  the 
programme  will  investigate  those  differences  and  suggest  ways  of  maximising  performance.  The  programme  is 
divided  into  three  stages: 

a)  'living'  cognitive  task  analysis  of  UK  Royal  Navy  sonar  operators  whilst  they  perform  target 
detection  and  classification  tasks; 

b)  report  on  initial  observations  and  development  of  an  experimental  protocol: 

c)  data-analvsis  and  report  on  the  experimental  results,  with  recommendations  for  training  and 
system  displays. 

1.3.2  The  aim  of  this  stage  is  to  begin  the  cognitive  task  analysis  of  novice  and  expert  operators  in  both  nations 
using  co-ordinated  data-collection  procedures.  Specific  objectives  are  to: 

a)  establish  options  for  data  collection: 

b)  agree  the  data  collection  programme  and  develop  a  co-ordinated  procedure  for  the  cognitive  task 
analysis: 

c)  begin  the  task  analysis  -  examining  the  decision-making  of  both  novice  and  expert,  sonar  and 
TMA  operators  in  the  RN  as  they  conduct  identification  and  classification  tasks. 

1.3.3  An  interim  report  { I }  described  initial  progress  with  the  programme  design  and  set-up.  The  aim  of  this 
report  is  to  summarise  the  progress  to  date  and  the  progress  with  the  task  analysis,  in  particular.  Specific 
objectives  for  this  report  are  to: 


a)  describe  the  planned  programme  of  data-collection  and  analysis: 


l 


b)  describe  the  agreed  procedure  for  the  cognitive  task  analysis; 

c)  describe  the  early  results  of  examining  the  decision-making  of  novice  and  expert,  sonar  and  TMA 
operators; 

d)  identify  any  issues  for  programme  development  and  adjustment; 

e)  outline  the  next  steps. 

1.3.4  Later  stages  of  the  programme  will  include  presentations  to  NUWC  and  ONR  on  the  results  of  the  work 
and  the  Pis  will  produce  a  joint  report.  Wider  publication  -  in  other  defence  fora  and  in  non-defence 
media  -  will  also  be  pursued. 

1.4  Approach 

1.4.1  In  order  to  provide  a  full  picture  and  to  be  able  to  accommodate  the  differences  between  individuals, 
systems  and  nations,  the  study  will  analyse  operator  performance  from  several  different  perspectives: 

a)  an  Organizational  analysis 

The  study  will  examine  background  factors  in  the  two  services,  to  identify  the  critical  factors  that 
determine  performance  and  to  agree  ways  of  describing  or  measuring  them; 

b)  an  Observational  analysis 

The  study  will  include  examination  and  observation  of  both  expert  and  novice  Sonar  and  TMA 
operators  during  target  detection  and  decision  making.  Operators  will  be  observed  at  individual 
tasks  and  in  team  settings: 

c)  Decision  analysis 

Comparative  Cognitive  Task  Analysis  (CTA)  is  a  principal  feature  of  the  study.  Applied  Cognitive 
Task  Analysis  (ACTA)  will  be  used  by  both  nations  to  capture  the  cognitive  aspects  of  task 
performance.  This  will  be  augmented  by  other  techniques.  For  example.  Job  Process  Chans  may 
be  used  to  highlight  the  contribution  of  man  and  automation  and  of  different  team  members  to  the 
tactical  decision: 

e)  Experimental  Analysis 

Expert  and  novice  operators  will  be  observed  and  interviewed  and  their  performance  will  be 
measured.  It  is  also  hoped  to  measure  perceptual  processing  during  decision  making  in  a 
controlled  task  and  to  measure  hean-rate  during  target  detection: 

f)  Military  analysis 

The  final  part  of  the  study  will  identify  ways  to  translate  the  research  findings  mto 
recommendations  for  best  practice  for  training  and  skill-development. 

1.5  Methodology 

1.5.1  Co-ordination  between  the  US  and  UK  studies  is  an  essential  feature  of  the  programme  and  this  has  been 
achieved  in  several  ways.  Early  visits  identified  the  areas  for  common  and  complementary  endeavour.  The 
NICOP  research  proposal  was  developed  jointly  {2}.  Documents,  such  as  the  management  plan  for  the  UK 
study,  were  exchanged:  and  regular  progress  reports-  were  provided.  Study-resuits  and  draft  reports  have 
also  been  exchanged.  Joint  visits  by  the  UK  and  US  Pis  to  a  number  of  VloD  and  RN  sites  have  helped 
develop  a  shared  frame  of  reference. 

1.5.2  An  important  aim  was  to  begin  data-collection  as  soon  as  possible  so  that  the  constraints  and  the  technical 
requirements  (of  the  USN.  RN.  NUWC  and  DERA)  would  be  identified  early  in  the  process  and  effective 


procedures  established. 

1.5.3  The  key  aspects  of  the  methodology  include: 

a)  task 

b)  subjects 

c)  performance  measures 

d)  CTA  method 

e)  biodata. 

1.5.4  Each  of  these  aspects  is  considered  in  turn  below. 

Task:  the  tasks  need  to  be  carefully  selected  to  fit  in  with  ongoing  training  facilities  and  to  accommodate 
differences  between  the  following  factors:  UK  &  US.  sonar  and  TMA,  novice  and  expert.  Task  differences 
will  be  addressed  by  identifying  the  activities,  operating  conditions,  performance  standards  and  associated 
skills  &  knowledge  that  are  required  of  operators  in  each  rank  and  position.  As  this  could  be  an  important 
factor  in  the  development  of  expertise  and  performance,  national  differences  will  be  compared.  For  example, 
the  two  nations  have  very  different  policies  on  the  balance  between  sea  and  shore  training  and  this  needs  to  be 
taken  into  account. 

Subjects:  a  broad  ranee  of  subjects  will  be  made  available.  The  RN  will  provide  access  to  sonar  and  TMA 
experts  and  novices  at  all  rates  -  as  individuals  and  in  teams.  Permission  has  been  given  for  the  Pis  (Dr 
Masakowski  and  Mr  Hardinge)  to  observe  expert  sonar  operators  in  action  in  command  team  trainers  and  at 
sea  -  if  circumstances  allow.  It  is  recommended  that  full  use  should  be  made  of  the  extended  UK  facilities  on 
offer  as  part  of  this  programme. 

Performance  measures:  a  range  of  objective  and  subjective  performance  measures  will  be  used.  These  could 
include  measures  of  task  outcome,  records  of  system  interaction,  team  interactions.  Instructor  assessments  and 
trainee  ratings.  It  is  hoped  also  to  use  physiological  measures.  Masakowski  has  measured  the  heart-rate  of 
sonar  operators  during  testing  sessions  {3}.  If  appropriate,  such  measures  might  be  used  in  this  study. 

CTA  method:  This  is  a  key  part  of  the  research  programme  and  the  method  needs  to  be  standardised. 
Chipman  {4}  highlights  the  important  features  of  the  methodology  that  should  be  identified.  These  include: 

a)  selection  of  tasks  that  merit  detailed  attention: 

b)  study  of  written  materials  and  training  materials  to  obtain  a  familiarity  with  the  job  and  specialised 
vocabulary  (this  is  especially  important  for  a  cross-cultural  study  of  this  sort): 

c)  selection  of  an  unstructured,  inclusive  approach  in  the  early  stages  to  capture  the  factors  that  are 
important  to  the  operators  (this  is  especially  relevant  for  a  study  like  this,  which  has  adaptive 
training,  based  on  an  expert-system,  as  the  long  term  product): 

d)  selection  of  structured  technique  (e.g.  goal/method  graphs)  to  identity  the  rules  and  their 
organisation: 

e)  abstract  nature  of  the  knowledge  involved  (i.e.  type  of  knowledge  representation): 
f|  basjs  for  selection  of  the  CTA  method  (rationale  or  principles  behind  the  method): 
a)  differences  between  understanding  and  approaches  of  expens  and  novices: 

h)  information  about  mental  models  of  the  task  and  the  team  context  of  the  work: 

0  information  about  the  expert's  mental  organisation  of  the  structures  and  functions  of  the  equipment. 


1.5.5  These  requirements  are  being  met  in  the  following  ways: 

a)  target  identification  and  classification  (for  sonar)  and  target  identification  and  location  of  targets 
(for  TMA)  are  the  selected  skills  for  analysis,  it  is  hoped  to  generate  common  target  data  so  that 
the  researchers  can  identify  and  compare  the  responses  of  the  different  national  groups; 

b)  familiarisation  with  the  jobs  and  their  specialised  vocabulary  was  deemed  especially  important  for 
a  cross-cultural  study  of  this  kind.  Training  syllabuses  will  be  exchanged:  doctrine  and  standard 
operating  procedures  (SOPs)  will  be  compared:  both  the  UK  and  US  Pis  observed  and  interviewed 
UK  operators; 

c)  the  analysis  of  critical  factors  affecting  performance  is  being  used  to  understand,  to  contextualize 
and  to  prioritise  efforts.  The  UK  as  well  as  the  US  researchers  used  the  Applied  Cognitive  Task 
Analysis  (ACTA),  (Militello  et  al,  1999  {5}).  This  method  is  flexible  and  inclusive  and  is  well 
suited  to  accommodate  the  kinds  of  contextual  factors  that  are  important  to  these  tasks  and  the 
differences  that  are  a  feature  of  this  comparative  study.  It  focuses  on  the  decision-making 
processes  and  on  the  cognitive  structures  that  individuals  use.  Ensuring  that  the  principal 
investigators  (Pis)  conduct  some  analyses  together  in  the  UK  will  further  develop  standardisation 
between  UK  &  US  researchers*  views  and  judgements.  Approaches  and  interpretations  will  be 
discussed  and  harmonised: 

d)  the  use  of  ACTA  will  be  complemented  by  more  structured  techniques.  Techniques  such  as 
■COGNET  {6}  and  MacSHAPA  {7}  are  being  considered.  One  graphical  method  being  considered 
is  Tainsh's  Job  Process  Charts  (JPCs).  This  provides  a  simple  way  of  identifying  the  separate 
contributions  of  man  and  machine  to  the  task  processes  (Tainsh  1982  )  {8}.  As  the  US  and  UK  use 
different  systems  and  may  have  different  degrees  of  dependency  upon  man  and  machine  systems  or 
upon  individual  and  team  processes  to  achieve  similar  tasks,  this  should  be  a  useful  way  of 
highlighting  important  differences.  This  should  provide  a  systematic  way  of  representing  technical 
knowledge  and  operators’  knowledge  structures.  Cognitive  Work  Analysis  (Vicente  1999  (91)  is 
also  being  considered.  Cognitive  Work  Analysis  (CWA)  provides  a  broader  view  than  CTA  and 
may  be  useful  for  capturing  the  relationships  that  lie  beyond  the  individual  operator  and  for 
highlighting  the  different  ways  in  which  function  is  allocated  within  the  Command  team: 

e)  the  abstract  nature  of  the  knowledge-domain  is  specifically  addressed  by  the  CTA  methods  being 
used  (i.e.  ACTA.  JPCs.  etc.): 

f)  differences  between  understanding  and  approaches  of  experts  and  novices  will  be  addressed  by- 
examining  operational  procedures,  the  performance  of  trainees.  Instructors  and  by  observing 
expert  operators  conducting  tactical  scenarios: 

g)  approaches  that  systematically  collect  data  on  individuals'  task  goals  and  objectives  will  be 
considered  as  a  way  of  measuring  situational  awareness  and  shared  mental  models  (see  Pearson 
1999  {10}): 


h)  examining  the  SOPs.  using  JPCs  to  highlight  individual  and  team  processes,  and  observing  expert 
operators  conducting  tactical  scenarios  will  provide  information  about  the  team  context  of  the  work 
These  methods  will  also  provide  information  about  the  expert's  mental  organisation  of  the 
structures  and  functions  of  the  equipment. 

Biodata:  factors  such  as  age.  experience,  educational  background.  IQ,  sea  experience,  may  be  relevant. 
Tactical  doctrine  and  SOPs  may  also  present  some  important  national  differences.  These  will  be  considered 
and  military  experts  used  to  assess  or  scale  their  relative  importance. 
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1.6  Structure  of  the  Report 


1.6.1  The  next  section  describes  the  progress  made  with  setting  up  the  programme.  Then  progress  with  the  data 
collection  and  the  results  obtained  so  far  are  described  in  section  3.  Section  4  identifies  some  of  the 
background  factors  that  might  affect  performance.  Issues  to  be  addressed  in  the  later  stages  in  the  programme 
are  described  in  section  5.  The  plans  for  the  next  stages  of  work  are  detailed  in  section  6.  Conclusions  and 
recommendations  are  given  in  section  7. 

2  Programme  Design  and  Set  Up 

2.1  Co-ordination  between  the  US  and  UK  studies  was  an  essential  feature  of  the  programme  and  this  was 
achieved  in  several  ways.  The  NICOP  grant  was  based  on  a  joint  proposal  developed  by  the  Pis  (Dr 
Masakowski  and  Mr  Hardinge).  Two  early  visits  by  Dr  Masakowski  to  the  UK  (in  November  1998  and 
November  1999)  made  it  possible  to  develop  an  effective  joint  research  programme.  The  Pis  visited  all  of 
the  appropriate  RN  agencies  to  brief  them  on  the  goals  and  requirements  of  the  project  and  to  establish  the 
basis  for  future  interviews,  data  collection,  etc.  Details  of  the  authorities  and  individuals  consulted  are  given 
at  Appendix  B.  Briefs  on  progress  were  provided  to  staff  at  the  London  office  of  ONR  Europe  {11}. 
Project  planning  documents  (e.g.  timetables,  and  protocols)  were  exchanged  and  basic  data  (e.g.  CTA 
results)  were  exchanged.  In  these  and  subsequent  visits.  Dr  Masakowski  has  been  able  to  interview  RN 
experts  and  examine  system  interfaces  and  training  devices. 

2.2  Activities  in  the  early  stages  of  the  task  consisted  mainly  of  liaison  with  NUWC  and  staff-work  with  agencies 
in  the  UK  and  US.  The  main  areas  of  development  included: 

a)  the  technical  requirement: 

b)  contractual  arrangements: 

c)  MoD  approval  for  this  form  of  international  collaboration: 

d)  MoD  approval  for  this  line  of  technical  research  to  proceed; 

e)  CINCFLEET's  agreement  to  make  specific  RN  facilities  available. 

n  ^  DERA  (CHS)  has  now  obtained  approval  from  MoD  to  proceed  with  this  form  of  collaboration  and  from 
CINCFLEET  for  access  to  appropriate  facilities.  Because  the  training  and  experience  of  sonar  and  TMA 
operators  in  the  submarine  and  surface  fleets  is  very  similar,  the  RN  has  provided  access  to  both  groups. 
Arrangements  have  been  made  to  observe  and  interview  trainees  and  instructors  at  the  School  ot  Maritime 
Operations  (SMOPS).  HMS  DRYAD  and  at  the  Royal  Naval  Submarine  School  (RNSMS).  Agreement  has 
also  been  given  for  Dr  Masakowski  and  Mr  Hardinge  to  observe  expert  sonar  operators  in  action  in  the 
command  team  trainer  at  HMS  DRYAD  and  submarine  teams  in  the  command  team  trainer  at  Faslane  or  at 
Plymouth.  Also,  the  RN  has  suggested  that  it  should  be  possible  for  the  Pis  to  observe  teams  at  sea. 

4  The  technical  programme  has  been  developed  and  refined  through  liaison  with  Dr  Masakowski.  This 
established  kev  details  of  the  data  collection  programme.  In  broad  terms,  the  number  and  range  ot  interviews 
(i.e.  sonar  and  TMA.  surface  and  submarine,  novice  and  experienced  operators,  trainees  and  instructors)  were 
decided.  The  style  and  content  of  the  interviews,  the  data  to  be  collected,  timetabling,  etc  were  clarified.  The 
available  sources  of  data  were  reviewed  and  procedures  for  co-ordinated  data  collection  were  established. 

2.5  Technical  and  experimental  arrangements  have  been  made  in  line  with  the  agreement  and  as  required  to 
satisfy  the  requirements  of  the  project.  Programme  set-up.  therefore,  has  been  completed. 


3  Progress  with  Data  Collection 

3.1  Analysing  Operator  Performance 


3. Li  Background  information  describing  the  training  and  development  of  operator  skiiis  and  experience  is  bein 
collected.  Factors  such  as  training  patterns,  team  structures  and  entry  standards  will  be  analysed  and  trainin 
materials  are  being  examined  in  detail.  The  way  that  operator  functions  are  allocated  within  the  command 
team  is  being  explored:  operating  procedures  and  team  processes  are  being  analysed.  Using  ACTA, 
interviews  are  being  conducted  with  individual  trainees  and  experts  to  identify  the  cognitive  skills  and  the 
cues  and  strategies  that  are  used  in  sonar  and  TMA  tasks.  Later  work  will  focus  on  common  tasks  and 
scenarios  and  will  address  performance  measures  and  physiological  measures.  In  the  UK,  operators  from  both 
submarines  and  surface  ships  are  being  examined. 


3.1.2  Cognitive  Task  Analysis  (CTA)  captures  the  cognitive  aspects  of  task  performance.  ACTA  was  the  method 
that  was  used.  Lutzhoft  &  Susi  1999  {12}  evaluated  the  ability  of ’practitioners'  rather  than  psychologists  to 
use  the  ACTA  tool  to  develop  a  cognitive  demands  table  (CDT).  They  found  that  ACTA  could  be  used 
successfully  bv  SMEs  -  but  that  help  was  needed  to  develop  the  CDT.  In  our  study,  ACTA  is  being  used  by 
psychologists  working  with  SMEs. 


3.1.3  How  a  USN  operator's  task  compares  with  that  of  his  RN  counterpart  has  to  be  established.  The  tasks,  skills 
and  knowledge  that  are  appropriate  to  each  level  of  operator  in  the  RN  will  be  identified  and  related  to  US 
practice. 

3.2  Initial  Results 

3.2.1  The  first  interviews  with  individual  operators  have  begun.  Using  ACTA,  interviews  with  both  expert  and 
novice  sonar  and  TMA  operators  are  being  conducted  to  identify  the  cognitive  structures  used  during  target 
detection  and  classification. 

3.2.2  As  the  task  is  very  complex  and  is  executed  as  a  team  activity,  the  allocation  of  tasks  within  the  submarine 
Command  Team  was  studied.  Appendix  C  describes  the  ty  pical  allocation  of  tasks  between  operators  in  a 
Trafalgar  SSN.  Comparable  data  from  the  USN  should  enable  us  to  identify  the  critical  aspects  of  team 
structure  and  role  allocation  that  might  be  different  in  the  US.  The  allocation  of  tasks  within  a  surface  towed 
array  frigate  was  also  considered  (see  Appendix  D).  This  highlights  some  of  the  separate  task  elements  that 
are  combined  within  submarine  operators’  roles. 

3.2.3  In  line  with  ACTA  procedures.  Task  Diagrams  are  being  elicited  from  novice  and  expert  operators.  Appendix 
E  gives  an  example  of  TD  data  being  elicited  for  operators  in  a  Trafalgar  SSN.  A  TD  is  also  being  developed 
for  operators  in  a  Type  23  Towed-arrav  Frigate  (See  Appendix  F).  The  normal  use  of  TDs  is  to  identify-  the 
cognitive  structures  that  are  used  in  a  job.  With  sonar  and  TMA  operators,  standard  operating  procedures 
(SOPs)  provide  a  detailed  formal  definition  of  the  minimum  set  of  steps  and  functions  that  should  be 
executed.  These  SOPs  are  normally  available  to  all  operators  during  the  task  (SOPs  for  TMA.  for  instance, 
are  detailed  in  CB8713  (2D)  {13}).  In  addition,  numerous  studies  have  identified  some  of  the  cues  and 
responses  that  have  been  employed  in  past  sonar  and  TMA  operations  (see.  for  example.  {14}).  In  our  study 
the  TD  will  not  only  be  used  to  identify  which  tasks  the  operators  feel  have  high  cognitive  demands  but  aiso 
to  identify  areas  where  an  individual’s  knowledge  and  use  of  procedures  is  different  from  SOPs.  The  results  of 
our  analysis  of  how  operators  approach  the  task  and  of  the  TDs  that  they  generate  will  then  be  compared  with 
the  results  of  the  US  interviews  from  the  parallel  study. 

3.2.4  Although  the  sonar  function  can  be  described  in  simple  terms  -  such  as  Detect.  Analyse  and  Classify-  -  the 
operator's  task  is  very  complex.  The  operator,  through  training  and  experience,  builds  up  a  store  of 
knowledge.  This  would  include  facts,  beliefs  and  hypotheses  that  relate  to  the  objects  and  events  (e.g. 
equipment,  command  team.  SOPs  and  target-data)  that  he  has  to  deal  with.  The  operator  updates,  maintains, 
and  accesses  his  store  of  knowledge  from  information  coming  in  to  the  command  system  through  sensors, 
intelligence  reports,  reports  from  other  members  of  the  team.  etc.  He  will  diagnose  the  available  information. 
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formulate  objectives  and  alternative  courses  of  action  for  achieving  them,  evaluate  the  alternatives,  choose  his 
action,  monitor  how  that  action  is  executed  and  evaluate  the  results. 

3.2.5  Data  for  a  Knowledge  Audit  is  being  developed  (Appendix  G  gives  a  simple  example  of  some  of  the  data 
venerated).  Again,  results  from  the  US  &  UK  studies  will  be  compared.  SOPs  identify  much  of  what 
operators  should  do.  Part  of  our  task  will  be  to  identify  what  operators  actually  do.  Operators  may  do  less 
than  is  recommended,  they  may  execute  tasks  in  a  different  sequence,  and  they  may  use  additional  cues  and 
responses.  They  may  follow  SOPs  exactly,  but  expert  operators  may  weigh  and  judge  items  of  information 
differently.  A  significant  part  of  our  study  may  not  be  eliciting  cognitive  strategies  but  identifying  which  cues 
and  strategies  are  used  by  effective  operators. 

3.2.6  The  scenario  for  a  Simulation  Interview  is  being  developed.  This  will  be  defined  in  terms  that  are  equally 
applicable  to  tasks  that  operators  from  US  and  UK  submarines  might  execute.  The  scenario  will  be  used  to 
provide  a  common  context  for  UK  &  US  CTAs  -  so  that  the  data  from  task  diagrams,  knowledge  audits  and 
cognitive  demands  tables  are  comparable.  Some  of  the  components  of  the  approach  used  by  Radtke  &  Frey 
{15}  to  develop  Sis  might  also  be  employed  to  give  a  suitably  rich  picture.  Klein’s  format  for  the  CDT  will 
be  amended,  as  he  recommends,  in  order  to  meet  the  specific  requirements  of  this  study.  The  changes  are  to 
serve  two  main  aims:  better  to  reflect  the  specific  content  of  this  study;  and  to  provide  a  common  format  for 
data  collection  and  reporting  that  will  facilitate  comparisons  between  the  results  of  the  parallel  studies.  So  far 
the  CTA  has  concentrated  on  identifying  the  content  of  the  operators'  tasks.  In  the  next  stage  CTA  will  be 
used  to  determine  interaction  effects  between  display  and  operator  performance 

4  Factors  Critical  to  Performance 

4.1  Discussions  with  SMEs  in  the  set-up  phase  of  the  programme  revealed  a  number  of  factors  that  could  affect 
UK  operator  performance.  The  visit  reports  { 16  Sc  17}  identify  a  number  of  these.  Examination  of  previous 
research  has  also  identified  a  number  of  factors.  A  study  by  Wylie.  Mackie  &  Smith  ( 1 8} examined  the  impact 
of  different  stressors  on  sonar  performance.  They  surveyed  the  stresses  arising  from  the  task,  the  physical 
environment,  supervision  and  personal  health  and  they  identified  the  extent  to  which  each  of  19  stressors 
affected  different  aspects  of  sonar  performance.  They  distinguished  between  vigilance  during  search,  visual 
information  processing  alter  contact,  auditory  information  processing  (after  contact)  reasoning/decision 
making  during  target  classification  and  perceptual-motor  processes  during  target  tracking.  Several  of  the 
factors^ that  they  considered  (displays,  workstation  design,  operator  overload  and  Command  pressure)  could 
also  be  of  relevance  to  our  study. 

4  2  The  next  stage  of  work  will  involve  a  systematic  analysis  of  these  factors  to  identify'  those  that  are  critical  to 
effective  performance.  It  is  important  to  determine  the  relative  impact  of  these  factors  for  the  RN  and  USN 
and  for  the  different  sub  groups  (i.e.  for  sonar  and  TMA  operators,  for  novices  and  experts).  Therefore  a  way 
of  describing  or  measuring  these  factors  will  need  to  be  established.  The  factors  so  far  identified  are 
summarised  at  Appendix  H.  Initial  data-collection  has  begun,  but  this  should  be  prioritised  and  standardised. 

4.3  Entry  standards  are  one  factor  that  might  be  important.  The  entry  standard  for  RN  sonar  operators  is  defined 
in  terms  of  the  RN's  Recruiting  Test  (RT)  and  of  educational  qualifications.  RT  correlates  very  strongly  with 
traditional  measures  of  IQ  and  the  minimum  standard  corresponds  to  an  IQ  of  106-1 10.  If  the  IQ  equivalent 
of  USN  entry  standards  can  be  identified,  it  will  be  possible  to  determine  whether  entry  standards  couid  be  a 
differentiating  factor. 

4  4  The  way  that  individual  operators'  inputs  are  utilised  by  the  command  team  may  be  an  additional  factor. 
Research  by  McDevitt  ;  1 9}  studied  a  number  of  submarine  sonar  teams  and  found  a  significant  relationship 
between  the  use  of  cognitive  procedural  rules  and  performance.  As  teams  became  more  expert,  they  used 
more  rules.  He  found  that  a  test  of  declarative  knowledge  (i.e.  knowing  the  rules)  did  not  predict  team 
performance.  However,  after  extensive  practice  (over  a  14-day  period)  teams  learned  how  to  use  more  rules  in 
their  TMA  and  this  enhanced  performance.  There  was  no  significant  difference  in  the  performance  scores  in 
Sonobuov  Localization  or  in  the  rules-use  in  this  type  of  scenario.  A  comparative  evaluation  of  rule-use  by 
UK  and  US  TMA  teams  could  provide  a  useful  contribution  to  our  study. 


Training  time  is  another  factor  that  might  be  important.  RN  training  patterns  may  differ  slightly  -  depending 
upon  the  background  and  experience  of  the  operator  and  the  equipment  fit  of  the  platform  to  which  he  will  be 
drafted.  A  representative  Branch  structure  and  training  pattern  for  sonar  operators  is  given  at  Appendix  f. 
Detailed  comparisons  will  be  made  between  the  training  patterns,  training  times,  input  standards  and  training 
performance  standards  for  the  USN  and  RN. 

Equipment  fit  and  functionality  are  also  likely  to  be  a  significant  factor.  The  RN's  main  sonars  are  2020  (hull 
mounted)  and  2046  (towed  array)  for  the  submarine  fleet.  Sonar  2050  is  the  hull  mounted  system  for  the 
surface  fleet.  The  user-friendliness  of  some  of  the  interfaces  may  also  be  a  significant  factor.  There  are 
significant  differences  between  different  RN  sonars:  2050  has  a  linear  axis  to  represent  the  compass  bearing. 
The  lofargram  for  active  sonar  is  represented  as  an  upward  waterfall.  The  way  that  the  passive  display  is 
represented  is  not  intuitive.  It  does  not  provide  a  traditional  circular  display  (with  ship's  own  position  as  the 
centre  point  and  with  a  surrounding  compass  rose  to  help  identify  bearings)  and  it  does  not  display  the 
lofargram  in  the  traditional  form  of  downward  waterfall.  RN  systems,  however,  have  a  Plan  Position  Interface 
(PPI)  which  provides  a  full  screen  display  that  operators  can  adjust.  It  is  understood  that  this  is  not  adjustable 
in  corresponding  US  systems.  Differences  between  the  USN  and  RN  systems,  and  their  likely  impact  on 
operator  performance,  will  be  examined  in  some  detail. 

Issues 

In  the  course  of  this  first  phase  of  the  study,  a  number  of  issues  were  identified.  These  indicate  some  of  the 
likely  (additional)  lines  of  development  in  the  next  phase  of  work  that  were  not  specifically  identified  in  the 
original  proposal.  These  include: 

a)  broadening  scope  of  the  study  to  identify  and  scale  the  impact  of  organisational  factors  in  the  RN 
&  USN  on  the  performance  of  operators; 

b)  pursuing  other  sources  of  performance  data  (e.g.  the  RN’s  passive  area  capability  training  (PACT) 
scores,  performance  scores  from  team  training,  performance  measures  from  exercises  such  as  Fleet 
Battle  Hotel); 

c)  developing  ACTA  (especially  the  format  of  the  cognitive  demands  table)  to  meet  specific 
requirements  of  this  joint  study: 

d)  measuring  perceptual  processing  during  decision  making  in  a  controlled  task  and  measuring 
heart-rate  during  target  detection; 

e)  pursuing  the  development  of  common  target/task  stimuli  to  use  with  US  &.  UK  operators  in  our 
study: 

f)  utilising  additional  CTA  methods  to  capture  some  of  the  complexities; 

g)  observing  UK  command  team  trials; 

h)  developing  tools  to  identify  the  cognitive  processes  that  should  perhaps  be  used  and  to  monitor  and 
highlight  those  that  are  used  and  to  relate  these  differences  to  operator  performance. 

The  Next  Stages 

The  work  should  enable  the  USN  and  RN  to  identify  reasons  for  differences  in  operational  effectiveness.  It 
will  focus  on  the  human  aspects  of  the  task  -  that  is.  what  operators  do.  how  they  do  it.  how  well  they  do  it. 
How  the  sonar  operators'  individual  inputs  are  integrated  into  the  command  team's  tactical  decisions  will 
also  be  considered.  The  technical  systems  (i.e.  hardware  and  software)  will  be  examined  to  be  able  to  put 
these  human  performance  aspects  into  context,  bur  this  is  not  the  main  focus  of  the  programme. 


62  The  work  so  far  has  simply  laid  the  foundations.  It  has  set-up  the  machinery  for  a  truly  collaborative  study. 
It  has  established  a  common  basis  for  data  collection  and  analysis  -  using  techniques  that  can  maximize  the 
comparability  of  results.  Initial  data-col lection  has  begun.  The  US  and  UK  Pis  have  been  able  to  conduct 
interviews  together  -  and  this  will  be  of  significant  factor  benefit  in  helping  them  integrate  their  parallel 
studies. 

6.3  Opportunities  to  examine  operator  performance  in  such  detail  and  to  be  able  to  conduct  this  kind  of 
international  comparison  are  rare.  The  work  aims  to  translate  research  findings  into  recommendations  for 
best  practice  for  skill-development.  It  will  identify  aspects  of  best  practice  and  will  provide 
recommendations  for  the  development  of  training  methodologies  and  display  designs.  The  pay  off  is  to 
reduce  training  time,  reduce  decision-making  time  and  to  enhance  operator  performance  for  both  groups. 

6.3  NICOP  sponsorship  of  the  UK  part  of  this  programme  has  provided  several  specific  benefits.  It  has 
generated  data  from  RN  establishments.  It  has  enabled  the  researchers  to  develop  a  co-ordinated 
experimental  programme  and  protocol  that  will  provide  data  from  both  navies  that  will  be  comparable.  UK 
involvement  during  this  phase  of  the  work  has  also  contributed  to  the  study  methodology.  Reviewing  the 
experience  of  RN  specialists  has  also  encouraged  the  study  to  consider  a  broader  range  of  critical  factors. 

6.4  The  thrust  of  the  work  in  this  first  phase  has  been  on  identifying  what  it  is  that  RN  operators  do.  This  work 
will  continue  in  the  next  phase  -  but  that  phase  will  also  focus  on  identify  ing  what  it  is  that  the  RN  operators 
do  well .  Collecting  measures  of  performance  will  therefore  be  an  important  pan  of  that  phase  of  work.  The 
opportunity  to  observe  RN  teams  working  in  the  command  team  trainers  and  at  sea  will  be  of  significant 
benefit. 

6.5  The  third  phase  of  the  programme  will  concentrate  on  identifying  the  performance  differences  between  USN 
and  RN  operators  and  on  determining  what  we  could  do  about  that.  The  work  here  would  include  the 
following  steps: 

•  compare  the  procedures,  the  systems  used  and  the  processes  of  skill  development 

•  relate  performance  levels  achieved  to  the  critical  differentiating  factors  identified 

•  analyse  the  data  to  determine  the  effects  of  training,  displays  and  decision  support  on  proficiency 

•  utilise  the  data  to  predict  operator  proficiency 

•  identify'  best-practice 

6.6  To  earn  the  work  forward  a  number  of  objectives  have  yet  to  be  realised.  As  identified  in  the  original 
proposal,  the  next  phase  of  work  is  to  include  the  following  activities: 

ai  the  Pis  will  each  provide  a  letter  report  giving  a  detailed  description  of  the  data  collected  from 
phase  l-  to  be  incorporated  in  a  joint  report: 

b)  they  will  identify  and  agree  a  list  of  the  factors  that  appear  to  be  significant  determinants  of 
performance  for  either  or  both  groups: 

c)  from  this  they  will  develop  a  co-ordinated  method  for  recording  the  factors  and  of  measuring 
their  importance  and  measuring  associated  outcomes  where  possible: 

e)  provide  a  systematic  analysis  of  expert  and  novice  perceptual  and  cognitive  strategies  during  a 
target  detection  task. 

6.7  In  line  with  the  requirements  of  a  NICOP  grant,  the  results  of  the  study  will  be  publicly  disseminated.  This 
work  will  be  presented  at  the  5,h  Naturalistic  Decision-Making  Conference  in  May  2000.  It  will  also  be 
presented  at  defence  forums.  A  presentation  will  be  given  to  the  next  meeting  or  the  LS/UK  International 
exchange  agreement  (IEA592S)  in  the  summer  of  2000  and  to  the  appropriate  Panels  in  The  Technical  Co¬ 
operation  Programme  (i.e.  Technical  Panels  2  &  9  (Training  Technology  &  Human  Factors  in  Warships)). 
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7  Conclusions  and  Recommendations  ; 

7.1  The  objectives  of  the  first  phase  of  the  programme  have  been  completed.  Programme  set-up  has  been 
completed  and  co-ordinated  and  experimental  procedures  have  been  established.  Background  data  are  being 
collected,  and  the  cognitive  task  analysis  of  expert  and  novice  sonar  and  TMA  operators  has  begun. 

7.2  It  is  concluded  that  as  well  as  individual  differences  in  operator  capability,  factors  such  as  team  processes 
may  be  a  significant  performance  factor.  Methods  to  analysis  these  are  being  developed  and  it  is 
recommended  that  as  well  as  examining  the  operators5  inputs,  the  study  should  address  the  way  that 
individual  operator  inputs  are  processed  by  the  command  team. 

7.3  It  is  anticipated  that  this  study  will  enable  indicators  of  best  practice  to  be  provided.  The  analysis  of  the  data 
on  the  issues  outlined  above,  will  provide  guidelines  on  aspects  as  diverse  as:  sonar  operator  entry  selection 
levels;  individual  operator  performance  levels;  training  content  and  delivery;  human  computer  interface 
design;  and  integration  of  the  sonar/TMA  process  to  command  team  decision  making.  It  is  therefore 
recommended  that  the  next  phase  of  the  programme  should  be  supported. 

7.4  The  response  from  the  RN  has  been  very  positive.  Discussions  identified  common  interest  in  other  areas 
relevant  to  operator  performance  such  as  task  automation,  improving  human  factors  integration,  Netcentric 
warfare  and  collecting  exercise  data.  It  is  recommended  that  full  use  should  be  made  of  the  extended  UK 
facilities  on  offer  as  part  of  this  programme. 

7.5  In  addition  to  the  required  deliverables,  it  is  recommended  that,  in  Phase  2  of  the  programme,  presentations 
on  this  work  should  be  made  at  the  next  Meeting  of  Information  Exchange  Agreement  5928  and  to  TTCP 
Panel  2  (Training  Technology)  and  Panel  9  (Human  Factors  in  Warships). 
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Glossary  of  Terms 


ACTA 

Applied  Cognitive  Task  Analysis 

ADAC 

Acoustic  Defence  Analysis  Centre 

AI 

Action  Information 

ASWSC 

Anti  Submarine  Warfare  Sensor  Co-ordinator 

Bio 

Biological  data 

CATMA 

Computer  Aided  Target  Motion  Analysis 

CDT 

Cognitive  Demands  Table 

CHS 

Centre  for  Human  Sciences 

CINCFLEET 

Commander  in  Chief  Fleet 

CO/XO 

Commanding  Officer/Executive  officer 

CPO 

Chief  Petty  Officer 

CSR 

Crank  shaft  rate 

CTA 

Cognitive  task  analysis 

DEMON 

Demodulation 

DERA 

Defence  Evaluation  and  Research  Agency 

DRIC 

Defence  Research  Information  Centre 

EFR 

Engine  firing  rate 

FOSM 

Flag  Officer  Submarines 

FOST 

Flag  Officer  Sea  Training 

HCI 

Human  Computer  Interface 

HMS 

Her  Majesty's  Ship 

JPC 

Job  process  chart 

KA 

Knowledge  Audit 

LOM 

Leading  operator  mechanic 

Met 

Meteorological  information 

MoD 

Ministry  of  Defence 

NICOP 

Naval  international  co-operative  opportunities  in  scienc 
and  technology  programme 

NUWC 

Naval  Undersea  Warfare  Center 

OJT 

On  Job  Training 

OM(AW) 

Operator  Maintainer  (Above  water) 

ONR 

Office  of  Naval  Research 

PI 

Principal  investigator 

PO 

Petty  Officer 

PO(R) 

Petty  Officer  (Radar) 

PSC 

Passive  Sonar  controller 

PSD 

Passive  Sonar  Director 

PSO 

Passive  Sonar  Operator 

PWO 

Principal  Warfare  Officer 

RN 

Royal  Navy 

RNSMS 

Royal  Naval  Submarine  School 

scoow 

Ship  Control  Officer  of  the  Watch 

SI 

Simulation  Interview 

SME 

Subject  MatTer  Expert 

SMOPS 

School  of  Maritime  Operations 

SOPs 

Standard  operating  procedures 

SR 

Sound  Room 

SVP 

Sound  velocity  profile 

TS 

Tactical  system 

TMA 

Target  Motion  Analysis 

TTCP 

The  Technical  Co-operation  Programme 

UK 

United  Kingdom 

Appendixes 

Appendix  A:  Terms  of  reference  of  the  study 


Cognitive  Task  Analysis  of  Decision  Strategies  NICOP  N9-04:  Task  Breakdown 


The  research  programme  will  be  conducted  in  three  phases  as  described  below. 

Phase  I 

Mr.  Hardinae  will  conduct  a  living  cognitive  task  analysis  of  UK  Royal  Navy  sonar  operators  whilst  they  perform 
target  detection  and  classification  tasks. 

This  phase  will  comprise  the  following  activities: 

a)  Establish  options  for  data  collection  (i.e.  subjects,  locations,  tasks,  etc) 

b)  Agree  a  co-ordinated  procedure  for  cognitive  task  analysis 

c)  Examine  the  decision-making  of  novice  and  expert  operators  conducting 
identification  and  classification  tasks  at  HMS  DRYAD 

d)  Examine  the  decision-making  of  novice  and  expert  operators  conducting 
identification  and  classification  tasks  at  HM  Naval  Base  Faslane 

Deliverable:  Agreed  data  collection  programme  and  procedure:  letter  report  summarising  the  process 

and  outcome  of  Task  Analyses. 


Phase  II 

Mr  Hardinae  (and  Dr.  Masakowski)  will  report  on  their  initial  observations  and  develop  an  experimental  protocol. 
This  phase  will  include  the  following  activities: 

a)  Mr  Hardinae  and  Dr.  Masakowski  will  each  prepare  a  report  of  their  observations,  to  be  incorporated  in  a 
joint  report 

b)  They  will  identify  and  agree  a  list  of  the  factors  that  appear  to  be  significant  determinants  ot  pertormance 
for  either  or  both  groups.  (These  factors  might  include  operator  characteristics,  task  features,  work 
processes.  HCI  features  and  an  understanding  of  system  functionality.) 

c)  From  this  they  will  develop  a  co-ordinated  method  for  recording  these  factors,  of  measuring  their 
importance  and  of  measuring  associated  outcomes  where  possible. 

Deliverable:  Letter  report  giving  a  detailed  description  of  the  data  collected  from  phase  l-  formatted  as 

a  chapter  for  a  combined  report. 

Phase  III  . 

Mr  Hardinse  (and  Dr.  Masakowski)  will  analyze  and  report  on  their  experimental  results.  Recommendations  .or 

training  and  display  will  be  developed. 

This  phase  will  include  the  following  activities: 

a)  Mr  Hardinge  and  Dr.  Masakowski  will  co-ordinate  the  methods  of  analysis  and  will  report  on  the 
experimental  findings 

b)  Lessons  learned,  best  practice,  recommendations  for  training  and  recommendations  for  sonar  display 
will  be  developed 

c)  vir  Hardinae  and  Dr.  Masakowski  will  exchange  the  results  with  the  NUWC  study-team  (subject  to 
each  participant’s  national  security  requirements). 

Deliverable:  (  Joint)  presentation  of  results  to  NUWC.  Inputs  to  joint  final  report  that  will  include 

recommendations  for  training  and  recommendations  for  sonar  displays 
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Appendix  B  -  Individuals  and  Establishments  Consulted 


WO  Carver 

Military  Adviser 

DERA  Sea  Systems, 
Winfrith. 

SSN  team  structure  and  task 
allocation 

Mr  J  Channel  1 

Training  Manager 

SMOPS,  HMS  DRYAD, 
Portsmouth 

Sonar  &  TMA  training.  CTA 
subject 

Cdr  C  Christie 

SFTO 

FOSM,  Northwood 

Cdr  Dalton 

MoD(N)  R  (Strat) 

MoD,  MB,  London 

Common  aspects  of  surface 
&  submarine  sonar  &  TMA 
performance 

Mr  Dunn 

CBT  Manager 

SMOPS,  HMS  DRYAD, 
Portsmouth 

Content  of  passive  sonar 
training 

Cdr  D  Forbes 

OC,  ADAC 

DERA,  Famborough 

Differences  between  UK  & 

US  operating 

CPO  Gowman 

Instructor 

SMOPS,  HMS  DRYAD, 
Portsmouth 

Sonar  &TMA  training,  CTA 
subject 

Lt  Cdr  Hayward 

DSWTO 

RNSMS.  Portsmouth 

UK  training  pattern; 

differences  between  UK  & 
US  Training  and  operating 


Cdre  F  Hiscock 

ACOS(W) 

Fleet  Warfightins  &  Trainins 
Brd.  C  INC  FLEET 

Programme  set-up 

Mr  T  Mansell 

Researcher 

DSTO 

Issues  &  focus  in  A  us  R&D 
programme  on  submarine 
sonar  &  TMA 

Lt  Cdr  K  Mardon 

SOASW,  SMOPS 

HMS  DRYAD.  Portsmouth 

UK  training  pattern 

Mr  Mike  McDevitt 

Researcher 

KAPOS  Assocs.  Inc.  USA 

USN  TRACE  programme 

Cdr  J  Pen 

SOI,  N7  Training 

FOSM.  Northwood 

Programme  set-up 

Mrs  E  Porteous 

Researcher 

DERA.  HMNB  Portsmouth 

Entry  standards 

Cdr  F  Price 

DOR(Sea)  R 

MoD.  MB.  London 

US  UK  Liaison 

Cdr  S  Robinson 

SOI  UWW 

NT.  CINCFLEET. 

Northwood 

Programme  set-up.  RN 
liaison 

Mr  G  Schwartz 

Researcher 

DSTO 

Focus  of  Aus  R&D 
programme  on  submarine 
TMA;  CWA. 

Dr  M  Tainsh 

Researcher 

DERA  (CHS),  Famborough 

HF  aspects  of  MoD 
submarine  sonar  R&D 

Dr  G  Walker-Smith 

R(C)2 

MoD/IRC.  London 

UK  US  R&D  liaison 

Prof  W  Weldon 

ONR 

ONR,  London 

Programme  set-up 

Mr  G  Vongas 

EC(AWB)Sc2.8 

MoD,  MB.  London 

US' UK  Liaison 

WO  White~ 

ASW1 

SMOPS,  HMS  DRYAD 

Sonar  TMA  training 

Mr  L  Yeo 

R(CS)I 

MoD/IRC,  London 

MoD  contracts 
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Appendix  C:  Allocation  of  sonar  activities  in  a  Command  Team  (Trafalgar  Class  SSN) 

CO/XO - (overall  decision  on  preferred 

solution) 


Watch  Leader - (advise  Command  on  evaluated  data) 


(shiphandling)  SCOOW 


L tl  CPO(TS) - (evaluation  of  captured  data) 

i 


2  x  LOM/OiVl(TS)  - (basic  picture  compilation  of  data) 


Control  Room - (Auto  &  manual  tracking  of  acoustic  data  -  via 

,  databus) 


Sound  Room 

*  Director  -  CPO(S) - Liaison  with  CR 

Sonar  Controller  -  PO(  S) - Manage  Operators 

Bow  -  2  x  LOM/OM  (S) 

Flank  -  1  x  LOM/OM(S) 

Analyser  (Narrowband)  -  2  x  LOM/OM(S) 

Active  Intercept  -  1  x  LOM/OM(S) 

Recorders  -  l  x  LOM/OM(S) 


*CPO(S)  (colloquially  CHOPS)  is  the  overall  director  for  the  compilation  of  the  sonar  picture.  He  is  free  to  move 
to  the  Control  Room  to  speak  directly  with  the  Sonar  Officer,  the  Watch  Leader  or  the  Command  as  necessary. 
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Appendix  D:  Allocation  of  sonar  activities  in  a  Command  Team  (Type  23  Frigate) 


- CO/XO- 


-(overall  decision  on  preferred  solution) 


OOW  (shiphandling) 


PWO  -J- 


-(advise  Command  on  evaluated  data) 


PO(R) - (evaluation  of  captured  data) 


2  x  OM(AW)  - (basic  picture  compilation  of  CATMA  data) 


Operations  Room - (Auto  &.  manual  tracking  of  acoustic  data) 


< — ASWSC  (direct  liaison  to - 

!  Command  for 

j  tactical  advice  & 

i  solutions) 


SDR 

PSD  -  Director 
PSC  -  Controller 
5x  PSO  -  Operators 
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Appendix  E:  Example  Task  Diagram  data  for  Passive  Sonar  Operations  (Trafalgar  Class  SSN) 


Establish 

routines 


Setting  up 

Check  configurations 
Consult  with  team 
-  Sonar  Controller 
-Vernier 
-Surveillance 
-Wideband 
-Broadband 
-DEMON 
-Active  Intercept 
-Daily  User  Checks 


Closing  up 

Sweeps 

Drills 

Procedures 


Report 
Contact  Info 


Report 

Changes 


Report 

Classification 


Screens  -  in  SR 

-sweep  beams 
-search  octaves 
-narrowband  content 
-Wideband  content 
-DEMON  content 
-Al 

Evaluate  in  context  of  intelligence, 
tactical  picture.  SVP  &  met. 
information. 


Changes  (e.g.  in  frequency) 
Investigate  changes  (e.g.  bearing) 


React  (to  Command  decisions) 


Control  Room 

•  Command  Classification 

•  Predict 

Bearing  Rate  Out  of  Turn ' 
Frequency  Out  of  Turn 


Collate  all  incoming  info 

-Met. 

-bio. 

-bearing 

-frequency 

Calculate 

-ERF 

-CSR 

-Gearing  ratios 
-No.  Shafts  &.  Blades 
-Shaft  Rate 
SR  Classification 


17 


Appendix  F:  Example  Task  Diagram  data  for  Passive  Sonar  Operations  (Type  23  Frigate) 


Establish 

routines 


Setting  up 

Check  configurations 
Consult  with  team 
-PSDs 
-Vernier 
-Surveillance 
-Broadband 
-Day  diary 


Closing  up 

Sweeps 

Drills 

Procedures 


Screens 

-sweep  beams 
-search  octaves 
-narrowband  content 
Evaluate  in  context  of  met.  etc. 


Initial  Record 

Contact 


Report 

Auxiliary  signature 
Propulsion  sources 


Report 

Operations  Room  (TMA) 
Reports  (3  min.  intervals) 


Initial  contact 


Changes  (e.g.  in  frequency) 
Investigate  changes  (e.g.  bearing) 


Collate  ail  incoming  info 

-Met. 

-bio. 

-calculate  bearing  &  speed 
-reduction  rates 
-blade  rates 


React  (to  Command  decisions) 
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Appendix  G:  Example  items  from  Knowledge  Audit  for  Passive  Sonar  Controllers 


Task 

Cues  &  Strategies 

Why  difficult 

-  Follow  set  drills 

-  Select  correct  frequency  ranges 
when  configuring  sonar 

Detect  target: 

-  Look  for  new  lines  at  known 

-  Lack  of  opportunities  to  gain 

Make  target  contact 

threat  frequencies 

experience 

-  Identify  changes  in  harmonic 

-  High  density  merchant  shipping 

structure 

area 

-  Identify  changes  within 

-  Bad  weather;  high  sea-state 

frequency  spectrum 

-  Problems  with  equipment 

-  Poor  intelligence 

-  Poor  management 

-  Poor  morale  within  sonar  team 

Predicting  sonar  range 

-  Predicted  sonar  ranges 

-  Area  of  search 

-  Expected  threat  directions 

-Time  allocated  tor  search 

-  Topography 

-  Mean  line  of  advance 

-  bottom  bounce 

-  Other  shipping 

-  upswelling  of  sound 

-  Land  masses 

-  Temperature;  Pressure;  Depth 

-  Ambient  noise 

profile 

-  Alertness  of  operators 

-  Direction  of  sea 

-  Good  beams 

-  Water  conditions 

-  Depth  of  towed  array 

-  Convergence  zones 

-  Cavitation  inspection  speed 

-  Array  flow  noise  limitations 

-  Own  ship  radiated  noise 

-  Array  length 

:  -  Array  safety  depth 

-  Track  hi  stop/ 
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Appendix  H:  Determinants  of  operator  performance 

1.  The  following  list  of  factors  was  identified  during  the  first  stage  of  the  programme.  They  are  offered  for 
consideration  during  the  next  stage  of  work.  A  systematic  review  of  the  factors  involved  and  a  process  for 
identifying  the  most  important,  from  both  UK  and  US  perspectives,  will  be  conducted  at  that  stage. 


Serial 

Factor 

Measure 

Importance 

Comment 

l 

Doctrine:  OPS 

2 

Team  structure 

-  #  and  structure 

3 

Allocation  of  tasks 

-  CWA  and/or  JPC:  study 

4 

Operator  characteristics: 

5 

Entrv  standards 

IO 

6 

Age 

-  Mean  age  at  each  sonar 
position  &  each  rate 

7 

Rate  or  rank 

-  Link  this  to  specific  tasks 

8 

Time  in  service 

-  Mean  No.  of  years 

9 

Time  at  sea 

-  #  of  tours  &  Mean  No.  of 
months 

10 

Exercise  experience 

-  -  of  ASW  exercises  &  recency 

11 

Equipment: 

-  type  of  sonar 

■ 

displays 

-  ergonomic  checklist,  user 
reports 

functionality 

-  SME  assessment 

May  be  classified 

14 

:  reliabilitv 

. "  ” 

"  "  M 

If 

Ran sze  &  accuracy 

M  f'  ” 

16 

DSS 

" 

ii  "  " 

i 

17 

Task  features:  shifts 

-  examine  rosters 

1 

18 

Work  processes 

-  examine  SOPs.  observe  teams 

19 

Training:  Training  time 

-  examine  timetables 

20 

Training  performance 

-  examine  Training  Objectives: 
analyse  training  progress  tests 

! 
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Appendix  I:  Branch  and  Training  Structure  of  Sonar  and  TMA  Operators  (Surface  Flotilla) 


WARFARE  BRANCH  SOURCE  BRANCH  PJT 


Cognitive  Task  Analysis  of  Decision  Strategies  -  Phase  2: 
What  do  UN  Operators  do,  and  how  well? 


Cognitive  Task  Analysis  of  Decision  Strategies  -  Phase  3: 
What  are  (he  performance  differences  between  USN  and 
UN  operators?  What  should  we  do  about  it? 
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Initial  Distribution  List 


External: 


Internal: 


Dr  M  McCluskey  ONR 

Lt  Cdr  D  Schmorrow  PhD  ONR 
Dr  Y  Masakowski  NUWC 

Dr  G  Matzke  ONR 


Dr  K  Chaplain 
Prof.  E  Farmer 
Cdr  S  Robertson 
Dr  G  Walker-Smith 
N  Hardinge 
A  Summers 


Business  Area  Manager,  CHS  3 

Technical  Manager,  CHS 

SOI  AWW  N7  CINCFLEET 

MoD/IRC 

CHS  3 

CHS  3 

DERA  (CHS)  Library  (2  copies) 

Defence  Research  Information  Centre  (2  copies) 
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Form  Approved 


REPORT  DOCUMENTATION  PAGE  OMB  No.  0704-0188 


The  public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  l  hour  per  response,  including  the  time 
for  reviewing  instructions,  searching  existing  data  sources,  gathering  and  maintaining  the  data  needed,  and  completing  and 
reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of 
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